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ABSTRACT 

As  COL  Jon  Moilanen  and  LTC  Donald  Craig  noted 
in  a  Military  Review  article  (May-June  2000,  pg.  13), 
“Leaders  must  possess  the  interpersonal  skills  necessary 
to  develop  and  sustain  high-performance  combined  arms 
teams,  as  well  as  work  with  other  services  and  nations 
during  mission  accomplishment...  These  teams  may 
routinely  deploy  as  part  of  a  coalition  to  locations  with 
immature  transportation  and  logistic  infrastructures  and 
uncertain  political  situations — conditions  requiring  high 
levels  of  innovation  and  cultural  awareness.”  This  paper 
discusses  an  innovative  training  prototype  that  not  only 
targets  the  development  of  interpersonal  ability  in  junior 
officers,  but  also  incorporates  many  of  the  situational 
elements  described  in  the  previous  quote.  This  paper  will 
discuss  three  studies  of  the  prototype,  as  well  as  lessons 
learned  about  implementing  interactive  training  software 
in  military  settings. 

1.  INTRODUCTION 

Many  training  packages  have  failed  to  capitalize  on 
the  full  potential  of  new  technologies  for  creating  student- 
centered  learning  environments.  As  a  result,  many 
training  tools  are  little  more  than  online  textbooks  or  are 
so  technology-driven  that  the  learner  is  treated  as  an 
afterthought  (Hakkinen,  2002;  Mayer,  2001;  Moreno, 
Mayer,  Spires,  &  Lester,  2001).  However,  interactive 
multimedia  can  be  leveraged  to  create  training  tools  that 
engage  the  learner  and  facilitate  training  transfer  to  new 
situations  (Mayer,  1996,  2001).  Effective  multimedia 
training  tools  possess  several  characteristics: 

1.  Training  should  include  immersive,  realistic,  and 
complex  cases  that  are  based  in  concrete  experience 
(Hannafin,  Hill,  &  Land,  1997;  Jonassen,  2002; 
Lesgold,  2001). 

2.  Material  should  be  both  verbally  (e.g.,  written  and 
audible  words)  and  visually  stimulating,  because 
individuals  process  verbal  and  visual  information 
through  different  channels  (Mayer,  2001,  2004). 


Learning  is  enhanced  when  both  channels  are  tapped 
concurrently,  because  the  trainee  is  able  to  connect 
the  verbal  and  visual  representations,  resulting  in  a 
coherent  and  elaborate  knowledge  representation 
(Mayer  &  Sims,  1994). 

3.  Allowing  trainees  to  control  the  order  and  flow  of 
information  can  help  trainees  reduce  cognitive  load 
(Mayer,  Dow  &  S.  Mayer,  2003). 

4.  Tools  should  pose  thought-provoking  questions  to 
cognitively  engage  and  orient  trainees  (Chi  & 
VanLehn,  1991;  Martin  &  Pressley,  1991;  Mayer, 
Dow,  &  S.  Mayer,  2003;  Willoughby,  Wood, 
Desmarais,  Sims,  &  Kalra,  1997). 

5.  Having  trainees  generate  questions  can  aid  in 
comprehension  and  active  processing  of  material 
(Rosenshine,  Meister,  &  Chapman,  1996). 

6.  Tools  that  use  pedagogical  agents  (e.g.,  computerized 
instructors)  encourage  trainees  to  view  training  as  an 
interaction  with  a  human  rather  than  a  machine, 
which  can  contribute  to  gains  in  learning  (Atkinson, 
2002;  Mayer,  Sobko,  &  Mautone,  2003;  Moreno  et 
ah,  2001). 

7.  Training  should  emphasize  thinking  processes  instead 
of  “correct  answers”  (Hannafin  et  ah,  1997;  Jonassen, 
2002)  because  many  real  world  problems  do  not  have 
right  or  wrong  answers,  only  better  and  worse 
approaches. 

In  conjunction  with  the  Institute  for  Creative 
Technologies  (ICT)  at  the  University  of  Southern 
California,  the  Army  Research  Institute  (ARI)  developed 
Think  Like  a  Commander — Excellence  in  Leadership 
(TLAC-XL),  an  interactive  training  tool  that  incorporates 
many  of  the  above  listed  characteristics.  TLAC-XL 
combines  Hollywood  storytelling  with  interactive 
computer  software  to  provide  trainees  with  an  educational 
experience  that  is  exciting,  immersive,  and  cognitively 
challenging.  TLAC-XL  consists  of  two  parts:  a  case 
study  and  an  interactive  computer  module. 

During  the  case  study  phase  of  TLAC-XL,  trainees 
watch  a  13 -minute  film  called  Power  Hungry’  in  which  a 
battalion  is  tasked  with  securing  a  site  for  a  non- 
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government  organization  (NGO)  to  distribute  food  in 
Afghanistan.  The  captain  (CPT)  in  charge  of  the  site  is 
confronted  with  several  problems,  including  unfavorable 
terrain,  tactical  disadvantages,  and  Afghan  warlords.  The 
Power  Hungry  film  is  unique  from  typical  case  studies 
and  training  videos  in  that  the  story  was  written,  filmed, 
edited,  and  performed  by  Hollywood  professionals 
(Figure  1).  The  result  is  a  film  that  achieves  the  quality  of 
a  major  motion  picture,  while  still  carrying  the 
substantive  content  of  an  educational  product.  The  case 
study  itself  encompasses  six  teaching  themes  identified  as 
important  leadership  issues  by  CPTs  on  the  faculty  at  the 
United  States  Military  Academy  (USMA):  mission 
clarity,  shared  vision  of  intent,  command  influence, 
setting  a  model  of  command,  cultural  awareness,  and 
respect  for  experience.  These  themes  represent  elements 
of  the  mission  that  needed  to  be  achieved  in  order  to 
attain  mission  success,  but  were  not  achieved  due  to  the 
poor  quality  of  interpersonal  interactions  among  film 
characters. 


Figure  1.  An  Afghan  warlord  expresses  his  anger  to  an 
American  Soldier  in  Power  Hungry. 


respect  to  the  teaching  themes  and  reactions  to  the  TLAC- 
XL  software. 


Figure  2.  In  the  upper  left,  CSM  Pullman  responds  to  a  trainee 
question  about  his  influence  on  CPT  Young.  The  computerized 
mentor  is  located  in  the  bottom  right. 


2.  STUDY  1:CAS3 

Two  classes  at  the  Combined  Arms  and  Services 
Staff  School  (CAS3;  n  =  26)  volunteered  to  use  TLAC- 
XL  as  part  of  their  lesson  on  ethical  decision-making. 
CPTs  interacted  with  TLAC-XL  in  groups  of  two  to  five 
and  completed  surveys  at  three  points  during  training: 
before  the  film,  immediately  after  the  film,  and 
immediately  after  completing  the  interactive  module.  The 
majority  of  survey  items  were  on  a  seven-point  scale 
ranging  from  strongly  disagree  (1)  to  strongly  agree  (7);  a 
rating  of  4  indicated  neutrality  toward  the  statement. 


The  second  part  of  TLAC-XL  consists  of  an 
interactive  module  to  help  trainees  reflect  on  and  discuss 
specific  elements  of  the  Power  Hungry’  case  study.  In  the 
interactive  module,  a  computerized  mentor  leads  trainees 
through  a  sequenced  series  of  questions  designed  to 
stimulate  reflection  on  the  teaching  themes.  Additionally, 
each  main  character  embodies  a  teaching  theme,  and  the 
mentor  presents  trainees  with  an  opportunity  to  interview 
that  character  when  a  specific  teaching  theme  is  the  focus 
of  discussion.  Trainees  use  a  natural  dialogue  interface  to 
interview  characters,  and  the  interviews  allow  trainees  to 
discover  new  information  about  the  scenario.  Trainees  are 
able  to  both  hear  and  see  characters  speaking  their 
answers  (Figure  2). 

By  the  end  of  training,  trainees  should  emerge  with  a 
better  understanding  of  how  the  interpersonal  behaviors 
of  a  leader  are  intertwined  with  tactical  outcomes  and 
team  processes.  Trainees  also  should  have  built  a 
conceptual  model  that  indicates  how  interpersonal 
interactions  are  related  to  each  teaching  theme.  The  next 
sections  present  three  studies  that  examined  learning  with 


Unless  otherwise  noted,  one-sample  f-tests  were  used 
to  examine  whether  item  ratings  differed  significantly 
from  a  test-value  of  4  (neutral).  Overall,  results  suggest 
that  both  the  film  and  software  received  positive  reactions 
(Table  1).  CPTs  reported  that  the  film  was  involving, 
interesting,  and  realistic,  and  the  software  was  valuable 
and  easy  to  use.  However,  CPTs  reported  that  characters 
often  were  unresponsive  to  their  questions,  indicating  that 
the  natural  dialogue  interface  needs  to  better  map  student 
questions  to  character  answers.  Despite  problems  with  the 
natural  language  interface,  CPTs  reported  that  they 
learned  about  the  various  teaching  themes.  CPTs  reported 
that  the  training  gave  them  a  strong  understanding  of 
cultural  awareness  and  respect  for  the  experience  of  non¬ 
commissioned  officers,  and  reactions  toward  the  lessons 
of  shared  vision  of  intent  and  mission  clarity  also  were 
favorable.  The  themes  of  model  of  command  and 
command  influence  did  not  appear  as  salient  as  the  other 
four  teaching  themes.  This  finding  might  stem,  in  part, 
from  the  behavior  of  the  lead  character,  CPT  Young,  who 
served  as  a  poor  model  of  command.  While  the  case  study 
easily  stimulated  discussion  about  the  many  things  that 
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CPT  Young  did  wrong,  the  software  spent  relatively  less 
time  steering  discussion  toward  things  he  could  have  done 
to  be  an  effective  leader. 

In  addition  to  survey  items,  CPTs  completed  a  self- 
report  measure  of  arousal  before  and  after  the  film.  A 
dependent  samples  /-test  indicated  that  CPTs  reported  a 
higher  level  of  arousal  after  the  film  than  before  the  film, 
providing  further  evidence  that  the  film  was  emotionally 
engaging,  t( 24)  =  5.66,  p  <  .001. 


Table  1 

Means,  Standard  Deviations  and  T-values  for  Survey  Items 


Reaction 

M 

SD 

t(2S)  \ 

Film  1 

Involving 

5.46 

1.27 

5.86* 

Interesting 

6.15 

1.12 

9.80* 

Realistic 

4.96 

1.84 

2.66* 

Confusing 

2.15 

1.41 

-6.70* 

Boring 

2.04 

1.18 

-8.46* 

Software 

Training  Value 

5.03 

1.81 

5.43* 

Ease  of  Use 

4.95 

1.23 

4.80* 

Easy  to  Control 

4.56 

1.33 

2.64* 

Character  Responsiveness 

3.08 

1.63 

-3.54* 

Teaching  Themes 

Mission  Clarity 

4.59 

1.31 

2.81* 

Command  Influence 

4.00 

1.21 

0.00 

Shared  Vision 

4.64 

1.04 

3.86* 

Model  of  Command 

3.90 

1.31 

-.49 

Cultural  Awareness 

5.54 

1.12 

8.58* 

Respect  for  Experience 

5.21 

1.22 

6.18* 

*  p<  .05;  One-sample  t-test  against  a  test-value  of  4  (neutral) 


In  sum,  findings  from  Study  1  indicated  that  the  film 
was  immersive,  energizing,  and  realistic,  which  is  an 
important  feature  of  training  for  the  complexity  inherent 
in  the  real  world  (Hannafin  et  al.,  1997;  Jonassen,  2002; 
Lesgold,  2001).  Results  also  suggested  that  TLAC-XL  is 
appealing  software  that  will  aid  in  training  several  of  the 
leadership  themes  identified  as  important  to  missions 
overseas.  However,  TLAC-XL’s  natural  dialogue  feature 
encountered  difficulty  in  matching  students’  questions  to 
answers  stored  in  the  database  of  character  answers. 


3.  STUDY  2:  TWO  INTERACTIVE  PACKAGES 

It  should  be  noted  that  TLAC-XL  is  a  prototype 
created  for  research  purposes,  not  a  finalized  training 
product.  Because  of  its  prototype  status,  TLAC-XL  has 
several  limitations.  First,  as  noted  in  the  CAS3  study,  the 
natural  dialogue  interface  is  problematic  in  that  the 
software  often  misclassifies  trainees’  text.  Consequently, 


trainees  often  receive  feedback  from  the  mentor  and 
characters  that  is  non-responsive  to  what  they  typed  into 
the  computer.  Second,  the  amount  and  type  of 
information  that  trainees  access  about  the  scenario  is 
contingent  on  the  quality  of  the  questions  that  they  ask  the 
characters.  Thus,  many  trainees  may  not  access  vital 
information  about  the  scenario  because  they  do  not  know 
the  appropriate  questions  to  ask.  This  problem  is 
exacerbated  when  trainees  possess  limited  background 
knowledge  or  are  unable  to  identify  what  they  need  to 
learn  (Graesser  &  Olde,  2003).  Third,  complete 
exploration  of  all  possible  character  responses  while  using 
the  conversational  interface  is  time  intensive  and 
cognitively  demanding,  which  might  result  in  fatigue 
effects  and  decreased  motivation.  Fourth,  TLAC-XL 
forces  trainees  to  interview  characters  in  a  predetermined 
order.  The  lesson  is  sequenced  so  that  the  mentor  queries 
trainees  about  a  teaching  theme,  and  then  presents  a 
character  for  trainees  to  interview.  Once  trainees  have 
interviewed  a  character,  they  summarize  their  interaction 
with  the  character  and  then  proceed  to  discussion  of  the 
next  theme.  While  the  benefit  of  such  structure  is  to  focus 
trainee  attention  on  one  theme  at  a  time,  this  structure 
precludes  trainees  from  returning  to  a  character  once  an 
interview  is  completed.  This  one-shot-inteiviewing 
feature  makes  it  difficult  for  trainees  to  integrate 
comments  across  characters. 

A  second  interactive  prototype  was  created  using 
Microsoft  PowerPoint  (TLAC-PP)  to  provide  a 
comparison  technology  for  TLAC-XL,  as  well  as  address 
some  of  TLAC-XL’s  limitations.  PowerPoint  was  chosen 
as  an  alternative  technology  because  it  is  low-cost  and 
readily  available.  In  TLAC-PP,  the  case  study  is 
presented  as  a  series  of  pictures  from  the  film  with  the 
film  audio  running  in  the  background.  While  the 
presentation  of  the  case  study  might  not  be  as 
sophisticated  in  TLAC-PP,  the  interactive  module  offers 
some  advantages  over  TLAC-XL.  In  TLAC-PP,  trainees 
have  significantly  more  freedom  to  explore  characters  in 
any  order  they  choose.  Trainees  can  select  any  character 
they  want  from  a  “cast  of  characters,”  and  then  are 
presented  with  a  list  of  all  possible  questions  to  which  the 
character  has  the  answers.  Unlike  TLAC-XL,  trainees  can 
see  and  access  all  information  available  about  the  case 
study,  so  trainees  do  not  need  as  much  base  knowledge  to 
generate  relevant  questions  to  ask  characters.  TLAC-PP 
also  does  not  use  a  computerized  mentor  to  lead  trainees 
through  a  structured  discussion  of  the  teaching  themes. 
Instead,  mentor  questions  (minus  the  mentor)  are 
presented  as  a  series  of  discussion  question  slides  that  can 
be  accessed  at  any  point  during  the  interactive  module. 

Ultimately,  both  TLAC-XL  and  TLAC-PP  present 
approximately  the  same  information,  but  substantive 
differences  exist  with  respect  to  how  the  interfaces  shape 
trainee  exploration  of  teaching  themes.  TLAC-XL  draws 
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on  a  guided  discovery  and  natural  dialogue  approach  that 
focuses  trainees  on  one  theme  at  a  time.  Each  character 
represents  a  theme,  and  opportunities  to  interview 
characters  are  in  a  sequenced  order.  Conversely,  TLAC- 
PP  does  not  present  themes  in  a  predetermined  order,  but 
allows  the  trainee  to  freely  explore  characters  with 
minimal  structure  imposed.  The  benefit  is  that  scenario 
information  is  more  easily  accessed;  however,  without  the 
sequenced  presentation  of  teaching  themes,  trainees  may 
not  attend  to  information  in  a  manner  consistent  with 
training  objectives  (Mayer,  2004).  The  next  several 
paragraphs  present  research  that  compared  the  two 
training  prototypes  with  respect  to  several  criteria, 
including  trainee  reactions,  memory,  interactive 
efficiency,  and  learning. 

3.1  Method 

Thirty-three  lieutenants  (LTs)  and  36  CPTs  from 
three  FORSCOM  installations  were  randomly  assigned  to 
either  a  TLAC-XL  (n  =  35)  or  TLAC-PP  ( n  =  34)  training 
session.  No  significant  differences  existed  between  LTs 
and  CPTs  on  the  variables  of  interest,  so  their  data  are 
reported  together.  Training  sessions  consisted  of  one  to 
five  officers  gathered  around  a  laptop  computer,  with  a 
modal  group  size  of  three  (M  =  3.35).  Group  sizes  were 
reasonably  equivalent  in  the  two  training  conditions.  Most 
training  sessions  consisted  of  Soldiers  of  similar  rank. 

Trainees  first  watched  the  case  study  and  then 
completed  several  measures  that  assessed  reactions  to  the 
case  and  memory  of  scenario  details.  First,  trainees 
completed  the  Positive  and  Negative  Affect  Schedule 
(PANAS;  Watson  &  Tellegen,  1988);  trainees  indicated 
the  extent  to  which  they  experienced  20  emotions  during 
the  case  study  on  a  scale  of  1  ( very >  slightly  or  not  at  all) 
to  5  ( extremely ).  Positive  affect  (a=  .88)  was  the  mean  of 
10  positive  emotions;  negative  affect  (a=  .86)  was  the 
mean  of  10  negative  emotions.  Trainees  also  answered 
four  items  about  how  suspenseful,  realistic,  confusing, 
and  boring  the  case  study  was  using  a  seven-point  scale 
ranging  from  strongly  disagree  (1)  to  strongly  agree  (7). 
Additionally,  trainees  responded  to  an  item  about  how 
well  the  story  communicated  the  thoughts,  feelings,  and 
emotions  of  the  characters;  this  item  served  as  an 
indicator  of  how  well  the  medium  conveyed  the 
personality  and  complexity  of  the  characters.  After 
completing  reaction  measures,  trainees  completed  a 
memory  test  that  consisted  of  24  true/false  items.  In 
addition  to  true/false  response  options,  trainees  were  told 
to  select  a  “don’t  know”  option  if  they  had  to  guess  about 
the  truthfulness  of  a  statement.  A  25th  item  that  was 
neither  true  nor  false  functioned  as  a  manipulation  check 
to  ensure  that  trainees  viewed  “don’t  know”  as  a  viable 
response  alternative;  82.4  %  of  individuals  responded 
“don’t  know”  to  this  item.  Memory  was  operationalized 
as  the  percentage  of  the  24  items  answered  correctly. 


Once  trainees  completed  the  first  set  of  measures, 
trainees  engaged  in  the  interactive  module  and  completed 
a  second  series  of  measures.  Trainees  responded  to  items 
that  addressed  general  reactions  to  the  training  product, 
character  interview  features,  and  mentor/discussion 
question  features.  Items  again  were  anchored  on  a  seven- 
point  scale.  In  addition  to  reactions  to  the  software, 
learning  was  assessed  using  nine  open-ended  questions 
that  required  conceptual  integration  of  training  material 
across  the  leadership  themes.  Two  raters  independently 
scored  trainee  answers  and  later  met  to  reconcile  any 
scoring  discrepancies. 

Researchers  also  created  two  variables  that 
functioned  as  indices  of  interactive  efficiency.  First,  the 
number  of  unique  character  responses  heard  during  the 
interactive  module  was  summed  to  indicate  how  much 
information  was  accessed  during  computer  interaction. 
Second,  trainees  were  timed  with  respect  to  how  long  they 
spent  exploring  and  discussing  in  the  interactive  module. 

3.2  Results 

Independent  samples  t-tests  were  conducted  to 
compare  TLAC-XL  with  TLAC-PP  with  respect  to 
trainee  reactions,  memory,  interactive  efficiency,  and 
learning.  Results  indicated  that  reactions  were  more 
positive  for  the  film  than  the  PowerPoint  presentation  of 
the  case  study  (Table  2).  Trainees  rated  the  film  as  more 
realistic  and  suspenseful  and  less  confusing  and  boring 
than  the  PowerPoint  presentation.  Additionally,  trainees 
reported  that  the  film  did  a  better  job  at  conveying  the 
thoughts,  feelings,  and  emotions  of  the  characters,  t(67)  = 
2.80,  p  <  .01,  suggesting  that  character  personality  is 
better  communicated  through  the  film  medium.  With 
respect  to  affect,  the  film  (M  =  3.09,  SD  =  .73)  evoked 
more  positive  affect  than  did  PowerPoint  (M=  2.63,  SD  = 
.73),  t(61)  =  2.60,  p  <  .05,  but  no  significant  differences 
existed  between  the  film  ( M  =  1.91,  SD  =  .69)  and 
PowerPoint  (M=  1.67,  SD  =  .66)  with  respect  to  negative 
affect,  t(67)  =  1.44,/)  =  ns.  Overall,  these  results  suggest 
that  the  film  is  more  emotionally  engaging  than  the 
PowerPoint  presentation. 

With  respect  to  memory,  trainees  exposed  to  the  film 
( M  =  65.48,  SD  =  11.23)  remembered  more  details  about 
the  scenario  than  did  trainees  exposed  to  PowerPoint  ( M  = 
50.12,  SD  =  14.12),  f(67)  =  5.01  ,p<  .001.  Because  the 
same  audio  was  present  in  both  the  film  and  PowerPoint 
versions,  these  findings  suggest  that  complex  case  studies 
are  more  memorable  and  cognitively  engaging  when 
presented  as  a  dynamic  visual  presentation  rather  than  as 
a  series  of  static  pictures. 

With  respect  to  the  overall  training  packages,  trainee 
reactions  were  similar  in  both  conditions  (Table  2). 
Trainees  indicated  that  both  tools  were  valuable  and  easy 
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to  use.  In  both  training  conditions,  trainees  also  reported 
that  character  interviews  provided  them  with  insight  about 
why  the  mission  failed.  However,  TLAC-PP  performed 
better  than  TLAC-XL  in  two  respects.  First,  trainees 
reported  that  characters  were  less  responsive  to  their 
questions  in  TLAC-XL  than  in  TLAC-PP.  Second, 
trainees  rated  the  mentor’s  use  of  questioning  as 
significantly  more  annoying  than  TLAC-PP’s  use  of 
discussion  question  slides. 


9.19,  SD  =  3.16),  1(67)  ='  1.11,  p  =  ns.  Indeed,  when 
conducting  an  analysis  of  covariance  to  statistically 
control  for  the  amount  of  information  that  trainees 
accessed,  TLAC-XL  was  superior  with  respect  to 
learning,  F(l,66)  =  5.48,  p  <  .05.  Such  findings  suggest 
that,  if  mechanisms  could  be  emplaced  in  TLAC-XL  that 
would  ensure  trainees  were  exposed  to  important 
information,  the  benefits  of  posing  questions,  asking 
questions,  pedagogical  agents,  and  structured  exploration 
might  be  better  achieved. 


Table  2 

Means,  Standard  Deviations,  and  T-Values 
for  Reactions  to  the  Two  Training  Tools 


TLAC-XL 

TLAC-PP 

Reactions 

M 

SD 

M 

SD 

t(67) 

Case  Study 

Confusing 

1.91 

1.44 

3.03 

1.70 

-2.95* 

Suspenseful 

5.54 

1.25 

4.94 

1.10 

2.13* 

Realistic 

5.66 

1.21 

4.94 

1.21 

2.46* 

Boring 

1.89 

1.21 

2.56 

1.24 

-2.29* 

Characters  Had 
Personality 

5.37 

1.11 

4.59 

1.21 

2.80* 

Software 

Training  Value 

5.17 

1.47 

5.06 

1.13 

.36 

Ease  of  Use 

5.34 

1.24 

5.29 

1.40 

.15 

Easy  to  Control 

4.80 

1.28 

5.15 

1.40 

-1.08 

Character  Gave 
Me  Insight 

5.31 

1.55 

5.53 

.96 

-.70 

Character 

Responsiveness 

3.11 

1.53 

4.41 

1.58 

-3.47* 

Mentor  Was 
Annoying  a 

4.46 

1.48 

3.47 

1.37 

2.83* 

*  p  <  .05;  athe  word  “mentor”  was  replaced  with  “discussion 


questions”  on  the  survey  for  trainees  in  the  TLAC-PP  condition. 


On  average,  trainees  spent  more  time  in  the 
interactive  module  of  TLAC-XL  (M  =  46.66  minutes,  SD 
=  13.21)  than  in  the  interactive  module  of  TLAC-PP,  (M 
=  33.06  minutes,  SD  =10.38),  1(67)  =  4.75,  p  <  .001. 
Although  trainees  spent  more  time  interacting  with 
TLAC-XL,  trainees  did  not  access  as  much  information  in 
TLAC-XL  (M  =  20.02  character  statements,  SD  =  6.13)  as 
they  did  in  TLAC-PP  ( M  =  30.08  character  statements, 
SD  =18.36),  1(67)  =  -3.02,  p  <  .01.  Thus,  the  TLAC-PP 
format  appears  to  have  an  advantage  over  TLAC-XL  in 
allowing  trainees  to  access  more  information  about  the 
scenario  in  a  shorter  amount  of  time. 


3.3  Summary  of  Study  2  Results 

The  results  of  Study  2  suggested  several  things. 
Presentation  of  the  case  study  in  a  film  format  appeared  to 
be  superior  to  a  PowerPoint  format.  Specifically,  the  film 
was  emotionally  more  engaging,  as  evidenced  by  trainee 
interest  and  involvement  and  the  relatively  stronger 
experience  of  positive  affect.  The  film  also  appeared  to 
be  more  cognitively  engaging  than  the  PowerPoint 
format,  as  evidenced  by  better  memory  of  scenario  details 
and  reports  that  the  film  was  less  confusing  than 
PowerPoint.  Additionally,  the  story  presented  in  a  film 
format  was  reported  as  more  realistic  than  the  same  story 
presented  in  a  PowerPoint  format,  and  character 
personality  appeared  to  be  conveyed  more  strongly  in  the 
film  format. 

Several  interesting  findings  emerged  in  the 
comparison  between  TLAC-XL  and  TLAC-PP.  Both 
TLAC-XL  and  TLAC-PP  were  rated  as  valuable  and  easy 
to  use  tools.  Trainees  also  reported  that  the  character 
interviews  gave  them  insight  into  the  scenario,  even 
though  the  character  interrogation  features  were  very 
different  between  TLAC-XL  and  TLAC-PP.  With  respect 
to  character  interrogation  features,  trainees  reported  that 
the  characters  were  significantly  less  responsive  to 
questions  in  TLAC-XL  than  in  TLAC-PP.  This  finding  is 
not  surprising  for  a  couple  of  reasons.  First,  as  mentioned 
in  Study  1,  the  natural  dialogue  software  is  sometimes 
inaccurate  in  matching  trainee  questions  to  character 
answers.  Second,  TLAC-PP  explicitly  lists  the  questions 
to  which  the  characters  had  answers.  What  should  be 
more  surprising  is  that  character  responsiveness  in  TLAC- 
PP  was  rated  relatively  low  ( M  =  4.41)  given  that  the 
listed  questions  were  explicitly  created  to  match  character 
answers.  This  might  be  due,  in  part,  to  the  nature  of  the 
character  personalities  themselves.  Some  characters  were 
a  bit  cagey  or  defensive  in  their  answers,  just  as  real 
people  might  be  when  faced  with  accepting  responsibility 
for  a  disastrous  mission.  Another  explanation  for  the  low 
ratings  might  be  that  the  questions  that  trainees  wanted 
answered  did  not  appear  in  the  list  of  character  questions. 


Although  trainees  were  able  to  access  more  With  respect  to  learning,  TLAC  XL  and  TLAC-PP 

information  in  TLAC-PP,  trainees  did  not  learn  more  in  produced  the  same  results.  Given  that  TLAC-PP 

TLAC-PP  (M=  8.38,  SD  =  2.83)  than  in  TLAC-XL  (M  =  presented  the  same  amount  of  information  in  less  time 
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and  would  likely  cost  less  to  develop,  a  logical  conclusion 
might  be  that  TLAC-PP  is  a  better  training  tool.  However, 
several  things  should  be  taken  into  consideration  before 
adopting  this  conclusion.  First,  when  controlling  for  the 
amount  of  information  that  trainees  were  exposed  to, 
TLAC-XL  actually  resulted  in  better  comprehension  and 
integration  across  the  six  teaching  themes  than  did  TLAC- 
PP.  This  suggests  that,  if  TLAC-XL  could  take  a  more 
proactive  approach  in  ensuring  that  trainees  are  exposed 
to  more  character  dialogue,  then  TLAC-XL  would  be  able 
to  capitalize  on  the  benefits  of  using  questions, 
pedagogical  agents,  and  guided  discovery.  Conversely, 
some  of  the  aspects  of  TLAC-XL  might  be  incorporated 
into  TLAC-PP,  such  as  a  more  structured  approach  to 
presenting  material  and  compelling  trainees  to  generate 
questions  they  would  like  to  know  about  the  scenario. 
Second,  although  we  explored  results  on  a  knowledge  test 
and  found  no  differences  between  the  two  training 
packages,  the  real  question  is  which  training  package  is 
more  likely  to  result  in  changes  in  behavior.  Given  that 
the  film  is  more  memorable  and  emotionally  engaging, 
behavioral  transfer  actually  might  be  more  likely  with 
TLAC-XL  because  Soldiers  who  encounter  a  similar 
situation  in  a  real  mission  will  be  better  able  to  compare 
what  they  remember  about  the  training  scenario  to  the 
situation  in  which  they  are  currently  immersed. 

4.  STUDY  3:  THE  FILM  AS  A  CASE  STUDY 

One  of  the  most  appealing  aspects  of  TLAC-XL  is 
the  wide  variety  of  ways  in  which  the  training  software 
can  be  used  as  an  instructional  tool.  Although  the  original 
training  concept  was  that  a  computerized  mentor  would 
assist  up  to  four  CPTs  through  discussion  of  the  case 
study,  portions  of  TLAC-XL  can  be  used  in  the  context  of 
larger  training  audiences.  Specifically,  a  human  instructor 
can  lead  discussion  of  the  film  for  a  larger  training  group, 
and  the  instructor  can  either  focus  on  the  original  six 
teaching  themes  or  construct  new  lessons  around  different 
teaching  points.  Moreover,  the  incorporation  of  a  human 
instructor  allows  TLAC-XL  to  be  used  with  a  broader 
training  audience,  such  as  NCOs,  because  the  instructor 
can  tailor  discussion  to  the  knowledge,  skill,  and  ability 
levels  of  a  given  training  group.  The  1/25  ID  Stryker 
Brigade  Combat  Team  has  used  the  film  in  such  a  way  in 
several  of  their  Stryker  Leader  Training  Courses,  and  this 
section  reports  survey  results  from  the  March  2003 
Stryker  Leader  Course. 

Approximately  23  Soldiers  (primarily  Sergeants) 
watched  the  film  as  a  group.  After  watching  the  film, 
trainees  broke  into  four  groups  in  which  instructors 
facilitated  discussion  about  the  film  for  approximately  20- 
30  minutes.  After  discussing  the  film  in  small  groups, 
trainees  gathered  into  the  large  group  and  discussed  for 
several  additional  minutes.  When  discussion  was  finished, 


21  of  the  trainees  completed  surveys  about  their  reactions 
to  the  film  and  the  discussion. 

Overall,  survey  results  indicated  that  Soldiers  reacted 
positively  to  the  film  and  lesson.  Similar  to  Studies  1  and 
2,  trainees  reported  that  the  film  was  interesting, 
involving,  and  realistic  (Table  3).  Many  trainees  indicated 
that  participating  in  small  group  discussion  and  vocalizing 
their  opinions  was  useful,  and  indicated  that  the 
discussion  helped  them  to  learn  about  the  six  teaching 
themes.  As  in  Study  1,  cultural  awareness  and  respect  for 
experience  appeared  to  be  the  most  salient  lessons,  while 
setting  a  model  of  command  and  command  influence 
were  the  weakest  lessons.  In  addition  to  the  original 
teaching  themes,  trainees  reported  that  they  discussed  and 
learned  about  additional  topics  that  were  brought  up  in 
their  discussion  groups.  For  example,  trainees  reported 
that  they  learned  about  adaptive  thinking,  visualizing  the 
battlefield,  preparing  for  contingencies,  and  timing. 


Table  3 

Means,  Standard  Deviations  and  T-values  for  Survey  Items 


Reaction 

M 

SD 

t(20) 

Film 

Involving 

6.48 

.81 

13.95* 

Interesting 

6.52 

.87 

13.25* 

Realistic 

6.10 

.94 

10.18* 

Confusing 

2.71 

1.98 

-2.98* 

Boring 

1.33 

.73 

-16.73* 

Lesson  and  Discussion 

Value  of  Training  Tool 

5.95 

.92 

9.72* 

Value  of  Group  Discussion 

6.00 

1.27 

7.25* 

Unresponsive  to  Training  Needs 

2.62 

1.53 

-4.13* 

Matched  My  Learning  Style 

5.43 

1.29 

5.09* 

Original  Teaching  Themes 

Mission  Clarity 

5.71 

1.19 

6.61* 

Command  Influence 

5.19 

1.33 

4.11* 

Shared  Vision 

5.71 

1.27 

6.18* 

Model  of  Command 

4.90 

1.55 

2.68* 

Cultural  Awareness 

6.67 

.66 

18.56* 

Respect  for  Experience 

6.48 

1.08 

10.53* 

Additional  Teaching  Themes 

Adaptive  Thinking 

5.71 

.90 

8.71* 

Preparing  for  Contingencies 

5.90 

1.22 

7.15* 

Terrain 

5.43 

1.75 

3.74* 

Seeing  the  Big  Picture 

5.29 

1.31 

4.50* 

Timing 

6.19 

1.17 

8.60* 

Visualizing  the  Battlefield 

5.81 

1.17 

7.11* 

Using  Assets 

6.00 

1.14 

8.04* 

*p  <  .05;  Ratings  based  on  a  7-point  scale  ranging  from  strongly 
disagree  (1 )  to  strongly  agree  (7).  T-values  based  on  a  one- 
sample  t-test  with  a  test  value  of  4  (neutral). 
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Trainees  were  asked  to  indicate  what  could  be 
improved  about  the  training.  Approximately  17%  of 
trainees  indicated  that  they  wanted  more  time  allotted  for 
discussion.  Approximately  24%  wanted  more  scenarios 
to  discuss. 

Study  3  is  interesting  because  it  demonstrates  the 
flexibility  of  the  TLAC-XL  software.  While  Studies  1 
and  2  focused  on  the  TLAC-XL  software  in  its  entirety. 
Study  3  extends  TLAC-XL  beyond  its  original  training 
concept.  First,  this  study  indicates  that  a  human  trainer 
can  facilitate  discussion  in  the  absence  of  a  computerized 
mentor.  Moreover,  because  the  Power  Hungry’  film  is 
realistic  and  complex,  the  instructor  can  expand  the 
discussion  beyond  the  six  original  teaching  themes  to 
focus  on  immediate  training  needs.  Another  important 
aspect  of  Study  3  is  that,  because  of  the  film  quality  and 
the  skill  of  the  instructors,  training  can  be  broadened  to 
accommodate  enlisted  Soldiers.  Study  3  also  established 
that  training  could  be  applied  to  large  group  settings. 

As  a  side  note,  the  best  indicator  of  the  effectiveness 
of  TLAC-XL  with  the  Stryker  training  audience  is  that  the 
film  was  used  again  with  the  1/25  ID  in  August  2003. 
However,  in  that  implementation,  trainees  conducted 
discussion  in  a  large  discussion  group  for  a  longer  period 
of  time  (approximately  90  minutes).  The  film  also  was 
used  in  similar  manner  at  Fort  Wainwright  in  December 
2003. 


5.  GENERAL  DISCUSSION 

Across  the  three  studies,  the  film  appears  to  be  an 
optimal  mode  of  presentation  for  a  case  study  about 
interpersonal  interactions  and  leadership  issues.  The  film 
captures  the  interest  of  trainees  and  is  emotionally 
arousing  and  engaging,  realistic,  and  memorable.  More 
importantly,  the  film  is  based  on  interpersonal  issues 
actually  encountered  by  officers  who  were  deployed 
overseas. 

With  respect  to  the  interactive  module,  results  point 
to  both  positive  and  negative  features.  In  Studies  1  and  2, 
TLAC-XL  was  rated  as  a  valuable  and  easy  to  use 
training  tool.  Additionally,  in  Study  2  trainees  indicated 
that  they  thought  that  the  character  interviews  provided 
them  insight  into  the  scenario  and  the  reasons  for  mission 
failure.  However,  not  all  character  and  mentor  features 
were  rated  positively.  Although  previous  research 
supports  the  use  of  social  agents  in  interactive  technology 
(Atkinson,  2002;  Mayer,  Sobko,  &  Mautone,  2003; 
Moreno  et  al.,  2001),  the  results  of  Study  2  indicated  that 
trainees  thought  the  computerized  mentor  was  annoying. 

More  problematic  is  that  the  natural  dialogue 
interface  did  not  accurately  match  trainee  questions  to 


character  answers  in  the  software  database  (Studies  1  and 
2).  Unfortunately,  this  problem  is  common  in  training 
programs  that  attempt  to  incorporate  a  conversational 
interface  (e.g.,  Ryder,  Graesser,  Le  Mentec,  Louwerse, 
Karnavat,  Popp,  &  Hu,  2004).  In  Study  2,  TLAC-PP 
skirted  this  problem  by  presenting  lists  of  character 
questions  for  which  characters  had  answers.  Question 
lists  served  a  more  important  function  than  merely 
shaping  trainee  reactions,  however.  Question  lists  helped 
trainees  to  visualize  what  information  was  available  and 
access  that  information  in  an  expedient  fashion.  As  a 
result,  trainees  were  able  to  access  more  information  in 
less  time.  Only  after  statistically  controlling  for  the 
amount  of  information  that  trainees  accessed  did  TLAC- 
XL  emerge  as  a  better  tool  for  learning.  These  findings 
suggest  that  many  positive  elements  of  TLAC-XL,  such 
as  structured  sequencing  of  teaching  themes  and  forcing 
question  generation,  will  only  optimize  learning  to  the 
extent  that  the  training  tool  ensures  that  relevant 
information  is  provided  to  the  trainee.  One  possible 
solution  is  to  allow  the  mentor  to  intervene  on  behalf  of 
novice  trainees  who  do  not  ask  the  right  questions;  the 
mentor  could  tell  trainees  what  sorts  of  questions  they 
should  ask  characters  or  could  pose  a  few  character 
questions  himself.  Such  an  approach  would  not  only 
ensure  that  trainees  were  exposed  to  relevant  information, 
but  also  would  allow  the  mentor  to  serve  as  an  expert 
model  for  the  types  of  questions  that  should  be  asked  in 
making  sense  of  complex  and  uncertain  situations. 

In  Study  2,  TLAC-PP  allowed  for  greater  freedom  to 
explore  the  situation  than  did  the  structure  imposed  by 
TLAC-XL.  However,  the  lack  of  structure  inherent  in 
TLAC-PP  might  have  been  problematic  for  some  trainees. 
Results  indicated  large  variability  (i.e.,  SD  =  18.36 
character  statements)  with  respect  to  how  much  trainees 
explored  the  interactive  environment  in  PowerPoint. 
Some  trainees  did  not  explore  all  six  characters,  while 
other  trainees  virtually  clicked  on  the  majority  of 
character  responses  available.  Lack  of  exploration  in 
TLAC-PP  could  be  due  to  motivational  problems  or  the 
inability  to  correctly  identify  gaps  in  knowledge  that 
require  further  exploration  (Graesser  &  Olde,  2003; 
Mayer,  2004).  Thus,  trainers  must  weigh  the  advantages 
and  disadvantages  of  imposing  structure  with  the  freedom 
of  exploration  in  order  to  find  a  balance  that  will  best 
accomplish  training  goals  (Mayer,  2004). 

Study  3  presents  a  way  in  which  some  components  of 
TLAC-XL  can  be  immediately  put  into  practice  while 
problems  of  the  natural  dialogue  interface  are  addressed. 
Specifically,  trainers  can  use  the  film  as  a  case  study  to 
discuss  the  six  teaching  themes  or  other  issues  as 
identified  as  important  training  topics  (e.g.,  what  it  is  like 
to  conduct  a  stability  and  support  operation).  An 
additional  benefit  of  having  a  human  conduct  discussion 
is  that  the  trainer  can  tailor  the  discussion  to  fit  the 
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knowledge,  skills,  and  abilities  of  the  training  audience. 
To  that  end,  ARI  has  created  an  instructor’s  manual  for 
how  trainers  can  use  the  film  as  a  case  study  to  talk  about 
leadership  issues  (Zbylut  &  Ward,  2004). 
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